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PROBLEM TO BE SOLVED: To perform a voice process for 
making an object voice to be aligned correspond to an input 
voice in real time v^ith a small storage capacity, 
SOLUTION: Data obtained by analyzing an object voice in 
frame unit sectioned by specific time unit are stored and an 
input voice is also analyzed in frame unit sectioned by similar 
time unit. A hidden Markov model is generated according to a 
phoneme dictionary and the frame of an object person 
corresponding to the time of a frame of the input voice is 
specified in phoneme unit by using one-pass viterbi algorithm 
to match time relation. Data corresponding to the frame 
wherein the phonemes of the input voice and object voice are 
made to match with each other are composed. 



Copyright (C); 1998,2003 Japan Patent Office 



2/2 



Detailed Descriptions of the Invention: 



[0004] 

The present invention is aimed at resolving said problem and the object thereof is to provide 
a voice processing apparatus capable of executing real-time processing with a small storage 
capacity for voice processing of associating in time series a target voice with an input voice for 
temporal alignment, and a karaoke apparatus having the voice processing apparatus. 



[0059] 

As explained above, according to the present invention, it becomes possible to perform 
voice processing for associating in time series a target voice with an input voice, for the temporal 
alignment, using a small amount of storage capacity in the real-time processing. 
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(54) [:aiM6?>^«i] $^ieLSS«:tei^r/^:^^::^yS« 



(57) \.WM^\ 

^S-^fcyu-A iicMW^r hf-^ ^^jS^T 




C2) 



laA^w^^D^m^'^ ^ y - ^ ^ ^^VPM-^^ b L Tiuia 

e ^ tf r ;v U X A J: o r t^fi|j^# * y^^T ^ 
L/cf5i3A:^^#^<^)«,^^^cS'-5i^r SuiBA;^j« 



[1**354 ] m^M 1 iBt)t(D«^Mffi^atctei,^r . 
[if 5 ] 1 fBtg<D^^«Laiiacctei^r . 

[If 7 ] l»*:g 1 iBt££D#mffi^gtc*5t,^T . 
^bf -5 C i J: or nfc^^HS^fBS-r €> C i «: 
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[ it*:B 8 ] if 1 IBigcD#^53ta^a CC:te . 

Sal E^'ltfSf B'lt^mti. '^m^f-^ h ^ rVlx* 
JS^^^tc T ^ - ^ ^Ji^-r ^ c <L J: o r *2s5 

SulBttMa^^^^KCi. WiB«Si0^^SCc<i:-pT?f$ 

[^*Ji 1 2 ] fi*3i 1 iBtgcDi^^iiis^Sfc^ji^ 
-r^ :7 A<bfyfa^a^^cD^*?tJjc?^is-r'Sfl?(iB »; 

[ ft 1 3 ] it* i 2 1 a® CD#^MS^ c i ^ 
ml^T'y -Y ^ > h*St^. b/cluiBA:^?#^CD#^ 



[ if *« 1 4 ] m>m, 1 2 iBigcD-i-^SiiiJl^icij \. > 

li^-r ^7 w - A i — SSc 5 i± ^ C i « i T -5 



50 



[M*S 1 5 ] If *Ja 1 iB*gO#^MS^g^C:Joi.^ 
r. 

mjiatt^^^i^^^^s^^. fuiar ^ > h ^mmm 



(3) 



mms, 1 8 ] m^m 1 ? nm(m^mmmmc^^> 
im^m 1 9 ] if 1 Mm<D^^mmmm^m^ ^ 

ft 7" a? CD B#m ?iM ei SB 3 it ^ P K <!: . 

^nmi.xmt>^^mt}^m,t^m^^ct^mwitr 
Lxmm-Rmmmmm^om^mmm^i^^t^ct 

[0 0 0 1 ] 
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[0 0 0 2 ] 

^xm^^ ^ i (.^ /cWWStJJPjjO^Si^ $ nri^ ^ o 

tCf^M-r 5 B$raiE«^ tr ^ D P ;r ^ > ( Dyna 

mic Time WarpinqiDTW) ti,^ "^iMW^hflXX^^ 

mtsinjhm^wm^^^-'>t{jxm\^^x\.^^<Dx. m^. 

(Hidden MarkovModel : HM 

[000 3 ] 

mi> ^nmxmm^m^nwmm}i^^crj::h<Dx\ tt 
[0 0 0 4] ±mi^fcmm^nw^r^fc^^c 

tA:t3^^t^m^Mx?^m'^\^^?>^'^9m^. ^i>?5:l^ 
[0 00 5 ] 

mmx o r . m^^M^^W ^y\y-J>^miiLX^m 



mommy u-At-^hmm^ni> v-tyB^maxnm 



5 

I >XmmAtf'^^0:>^^^ ^• "7 y - ^ ^ V' > :^)im^it b 

[0 00 6] it^:S2 ccfaife(D^B^{3:. 1 iBKO 

xmm'^^^^^^h)v^-^ts^ct^w&tr^o m^m 

5 ^CiB*g<D|feB^&*. 1 iB*g(D*^Mii^SCC*5Ci 

[0 00 7] m>m 7 iCiBifiOD^B^iS. f«*31 1 IBtgO 

w^m^ iBisf ^ c t ^1*14 i -r ^ o tmm s latgo 

l^BJti. i»^«i8ai£cD#^Mil^Si^ctsc^r. fuiB# 

A^c ^ / ^' ^ ^ - ^ /^^f^T ^ c <h J: r >tt^ 6 n/c 

[0 00 8] m^m 9 iicnmommitt. m^m i latfeo 

<:^i^jf$^#©oc J: o xnm:^ntcimi)^^mimwm<D 

n'yct^^t'ir^. m^mi 0 (^c$m<Dmm^^. it 
^mumo'^^mmmmc^i^^x. mmi^tPMrn^m 
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-r^VifB^nm-r^ct^mmtr^a m^mi uaa 
momMu. m^'mii^m<D'^^mm^m^c:^\.>x. m 

[0 009 ] if *:s 1 2 tciatg^Dlfeesti. 1 ia»g 
tBA;f:i^^cDW*?a^c*[j£T ^ ^ u - a i mnm».^^ 

mmu. m^i2nm(o^^mmmmttcis\,^x. ftrta 
T >f ^ > H ^s^^ . L fcmn^xt}'^^(0'^micM 

it^t^yu-j^mf)K 3^W*i-^TStaiB*fm#^ 

XmMBMm^^oyy u-ASfccD:^;^^ u-M.m^mm'r 
m 1 2 tamcD^^Ma^a^cfca^r . mmr v y > b 

digits:. U tcm^Atf'^^coi^m^cKm'r ^yv- 

t-'SMfa y 3 >*«isr Aiii feii>?:ci^ 

. Huf Bm^31^J^^^© J: o T frrl BA;^;^^ 
-SMtB*f^S^<?>BU IB y - 3 > ^S^T ^y\y-2>.t 

mm^mt. Huf BT ^ > h ^s^^^^^^^ T ^ 
[0010] m^m 1 6 i,cmsk<Dmm^. m^m i iBa 

MlSccS-^i^r. Mia>i**#^ittfiBA:^#^<!:^*tl£ 

*#m<h-rSo lf*:®l 7 0ciBtfe<?:>^B9^i. 6 IB 

«t<7:)^^Ma^scc :te I ^ r . M la A;^ yu- 
mt. mmiMi^^i^^y\y-2.^^xwmk^mh 

-^r-^^T c <t ^#ifi[i T-s . mm 1 8 cciBifecT)^ 

B^5^. n^'miitm<Dm^mimmcis\.^x. wm^ 

[001 1 ] m^^. 1 9 ^ctm.(omA\-^. i latg 
(D^^m^mm:^mfL^tj'7:t^mmx:h-yx. mm^ 

50 «^-r-5ftr^-^^iBti"r'Sft-r--^iBtt^l£i. ItFfB 



(5) 

7 

^iHi^ LT tJi:t?-r ^ fJB;^j^i5<?: ^ C i ^J^mi 

[0012] 

[013] [1 . ^Ifeff^SscDlf^] 

^^>:? h : Target) (D^^iC'fHii ^ J: ^ ^Cl^^^^^tT^ 
[ 0 0 1 4 ] ^2tC*5l^r. -^-Y UJ. ^>OS^^b 

A^WF^fl^t?1iHlSl53ic,4i:;^jT'2>. A:^;^^e-^^mg^ 40 
t^J-^t^-ti. A:^#Wt-^Sv*:7i.'^AmfiT^Wf9ffl 

[0015] mm^^mmb s t^. ^^aE'r-ssMs (sp 

ectral Modeling Synthesis) ^W^tToTjE^y^^JS^^ 
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ssaffi (^)i^y':p^h'yAi^w. ^9t^)\^^y'7sy^ 

[0016] **^»iBaSP6 f*^ci¥ b < SiBJTS 

^»ia'[t^6^cfB'ttstlfc^g^-^||)i^#Mt'r. -e<D>' 

^y'y^ (HMM) ^Cj:-:>T^^M;^^^ff^^b. tfc®31 

me^- > Ji^^^t^ ^ffl . \m^^ s tf ^ - b'T ;^ >; XA ic 

[0 0 17] T^-f.^>hgisi o&s. ^^^n)^tfc«s 

mrt^^Ati^^(onf^^-Ai^^^iSk^L^. ^m^^^^A 

tf?gPtCfm$tl/cA^^^C?^SiS^jS5>i. ^->5r*-^ h 

^sjfspi uct3:. ^^5^>^u-A^tiT^ja«^^«$n/c 

nS B#ra y a > ( region) ^firlBa * tlfc^^ij 
[0018] 1 2 }i . A:;)^WF^©ii«^^3^<*:*f 

^^^^^m^xm^'^y - 0 1 3 t^cw^tiv. 
mrnmy - 0 ^^sr 1 3 [mtcrmmmm^^mmm 
y-v:i^^m:Oxmfcrj:^^mn^^m-^o tc^ 
x\ :mmim}im^*3im^=^m^tc^yt^mmx^ 
mf-^t^wMi 4ccti. MIDI r-^-pmmf 

^mm^tixid^. MIDI 'f-^^mmr'-^^cm. 

t)r-^f)^^m^m^^^m-r^^mi^^m^x\.^ 

So 5 ^tM 7 m-mi&y- V :t,WJM^ 1 3 

'^)SL/rxt:'-;^l 8*>6tH:^rSo C<OJ:'>ifC, mM 
mtf^^A ^ 1 ^Cfnl;0^^ri^Pif S <!:. SfcPi*cD§^:!i?W 

$nrc=*So 

[0019] [ 1 - 2 . #^SS]y<Ctc:. *:||3^?g^-Cfla 

i^s^^^ccri^t^ri^B^-rSo ^^^sts. i^mt^ 



[0020] [1-2-1. h^^lW^itfitCoi^ 
B$KM»^«T0^t?^>O. 2;X7G-^ 
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[0 0 2 1 ] (1) 

mi] 



^(w - m) 



C CC. sgn{sCn)}-+l:sCn)>-0,-l:s(n)<0, 
w: "7 U--A^ 

s : A:^«-^ 

[0 02 2 ] (2) b'^:y^x^-- 

*J<j:l>\ a«Sb:-^5^;&^6-?^SiJh:'^;/^->.cDX'^-CD2:??[Rj 

7i<th<DX^X> . ^M^C\t. "Fundamental Frequency E 
stimation in the SMS Analv5is"CP.Cano. Proceedinqs 

of the Digital Audio Effects Workshop, 1898) tCTwo 

Way Mismuch¥^^^<b 1/^3^?^$ '5>o 

[0 0 2 3 ] -^SfJt:- 5^Cp):0^6SiJ^tf ^Cm)^ 

<D fc: ^ X ^ —\t:X^^m,^n 5 o 

[SK2] 



20 



N 



30 



a 



n 



40 



fn: nmm(D^m^-^mmk 

Anax: SAr>>''U^^- F 

[0024]— ;5^. mn^v9-im:itf^h^m\:f^y'^(py^ 
(D b' 9^ X ^ t^y^CCD^raS n 

[|^3] 



N 



fk: k*ac?>?S'Jt'-^. 
A f k : k # a CD ^arj fc: - ^ Mlvgifc i ^ n fig b fcafj* 

ak: k#a<DSfJ^T>:7M;5^^-^ F 
[0 02 5 ]fieor. F-a?;i'X'7-ti;X^coJ:0tc;tc 

[0 02 6 ] [1-2-2'. #-^te]i?^ii!Sii. -en^n 

[0 02 7] LGBr>'l':3 UXAi&jyT^c^To 



50 ©jSm 



(7) 



(i+e) ix (1-e) <!:lie>r.ocD::i- 
-So CCX\ m«0. 0 0 1 il^^/M^C^ffl 

\ (k= 1.2. -,2') f)m^ti^o 
k' =argminkd (Xv x'k) 

cc-c. d (x. x\) ^ad^r^-com^ffi^^^ 

3) Saitft^T^r^l §^aDktCOl,iT. x\ = Q (x) 

(Di^^c. T-^r©-^^ F;l'^-fe>Fn-< FT^tf^^: 
[0028] [1-2-3. m^r'-^r<K('C^ W^r- 



a 



"3" rab-sor. 



a 



(Dim 

(ASPIRATION) fe2^*«fc^$nrc>^« -eur. ^ 
^ (siLEMCE) ti. H$Paw^ab;5^^ci^©r itfc*«i^c^^ 

[0 0 2 9 ] ustc^Tcfc^^c. ^mm^<om^^- 

[0 0 3 0 ] -l2ri?V> h k^i^^Mr;^:3^';XA^I:iyT 



10 



20 



30 



40 
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h ffir SI 53 ^ C i tc J; 0 3ti?> 6 n -5 . 
[^5] 



[0 0 3 I] — ^'>>i<;b^mfil*ti. ^j^S^c^Ti: 
[|g^6] 



[0 03 2] >f^-i/3 > 

F^t-r^o 

[0 03 3] [1-3. ^-y .y F:7 U-Alf^l^-y 



m^mCtcmWiSM (Frame) ^^fJi^HlL. 

[0034] i£Mmj&^t\±. m^mrnwi (pitch) 
B#racctB^-rs^-i^Ax^>:7'7&5^^$nrc^^ (t t 

1. t t 2. ) o 

[0 0 3 5] tC^X. ^fetCgjegb^^Ccfc^JC, 

^^^c< ^4>^^«^cg:^§nrc^a^:^^li^^cD>'u 



50 



13 

*tlSL. U-At t 6- t t 1 O:0i6«llSiSn'Sy- 
- ^ -e^^il;05 r ^ <i: ^5 (c ^j: S . 
[0 0 3 6] [2 . ^]!^?fl^cDi6fP]:^cc:. ^{iffl^lgcDtii 

[0 03 7 ] [2 - 1 , mmnmMWiiC. mmmmc-:> 
-5. s-r. -^-f ^A*#^^«TWft)ti^ (s 1 ) „ :a 

[0 0 3 8] ;^^c:. W^«:fei:?:;^*faai^d^cS-^i^ 

2) . >'>3i<;l'S^^t3n/ci^-'^'v><-^:foJ:C>*^^^ 
$ n /c fcK -^X ^ t: 1 / ^• ;^ ^ - T n i; X A 

{Cci:oTi:^^#^^^S'r-S (S3) . HMM^7';l/:fe 
J:CK 1 /SX ^ - If r Jl'::^ V XAiCOl:*r t3:fg^Cl¥ L < 

mm^^iy^^'^h (S4) . s^B$ra*sif/c^cw^ 

«,%^tcfJ:o/c7^)>^/^^^^*^Mr^ (S5) . nf^^^^i^^ 
5iiaB#^nc4c5tt'5>'U-A?:#^T&feCDt?*€>o *^ 

ife3^y^B$^j t m 1 . tm2 tm^i^. n^%^<om 

^?Ej^ttl. tt2 (t^fBT-So 

[0 0 3 9 ] X^^:;:7'S5(7)W^CC4ol^r. ff/c^C^^ 
«ggCc^co/ci«^L/tJS-^^^ ( S 5 : Y e s ) . U 

^A:^»:7>h^p^^^L (S6) . mr^^^^zy^^'^mn 

cD5fe3fi'-v^»rf -5 <S7) . y\y-Kt!^iyVt{l^. ^ 
ig^^t^!^<hurft!ilI0/t:7U--A^*iH^ $fecc|ft0j 

tcf5I:7[y-A^l:^fc3&^?r^-rffl[i%'So -eur. A:^^* 

JS?«ti»tffr^'-^i^jai6i^M^r^)ab (SS) . 3! 
i^a-^-yx^mf ^CchCCi:-^^ (3 9) ff/c?^j:#^ 

[0 04 0] iC^-C. X7^>y:7''S5CD«S^C*5lir. 

if /c j'^^*^;^^ ii^ hx\.>r^^>t mm /cis^ » ( s 

5 ; No) . :7U-A:^'i;>h*'<>^y^>hUr 

(s 1 0) . m^-^^^i^'^^yi^-^mmmm^m^ 

(S 1 1) . :^v^:^y^s^iiCmy^ho M^^m^hiil^ 
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^^^imf^i^^\ty U-A:^^>h^'r>'^y-^> h 

ur. B#pa3j<-f 1 1 K tt2 t^m^ 

€>o L/:O^L/. U-A t t 3©«^i^J^30^ "n" ?r»l 
"a" (D5fegi:7 P-A t t 6tcB#PBl;i<>r>i^^^S&3-tf 

10 [0041] [2-2. tof^cDPiffli^j^^c, m^sbmcte 

[0042] [2-2-1. A:^f#^55'W]^9 A:tl 

?}ffi§ti/c#^ff#ii. *ii:7--yx^^$cj:ot:jaig 

-A#©aay>'>jJ^;i/^ffii=>T. f^tci¥L < J: 
^ Jctfc^^^^^j^ $ n -S J: ^ & ^ . 

[0043] [2-2-2. Bn-e^t'n:7^7^jl'];X^c. 
lai Oi&#ML'3^«c;&5 6v linv;Vn:7^^;l/ (HMM) 

-SCQr. left to riqhtm(D^'f)\^^m\^^X\<^^o 

[0044] mmt^cisi^x. ifmtfi iti^hi ^^-r 
30 -6^1^ (iKP^^mm) ^ a , , tm-r. m i o ^c^tm 

tfc^©-^jl^-r^^t^^a..iaL.TC^So ^itm<D^ 

h;i/(?)-»*vuxt*^*^-ii:-sfii^ mmity>^< 
jmmum) ^b, (xt) iRr^ ^y^ji^A^c^i^r. 

[^7] 

r 

tj:h>tii^^mmx. c(DJ:':>tji^r'j\^^mti'^ji^^y'^ 

(HMM) tnUtiXX:^^^ imM&^X\t.. ^- 

mn^m--3<i>x. 01 o^c^TcI:^^cfns immtm 



C9) 



[0045] [2-2-3. r^^^>hpxtc. mi 1 

[0046] [2-2-3- 1. l/Sxt:'^-fc*T;l^:3 
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' bj{deitaENERGY)iOt) * ^ j{VOJCENESS)^t)' ^ jiENERGY){Of) 



l<t<T, l<J<t 



t:ti-^$n/cffi^^0^b^i43:A-c^urt>6 (0> 

A) „ m^\t. ^^J t m 1 -fy^h^mi t m 3 ^ T'CDtlffl^ 
4:^^ "Si 1 e n c e" t^him "n I " ^(D 
J^:;^timJ&in^M^U. tfc^ "Si 1 e n c e" 
i:^^ **S i I e n c e " ^(D^^TstmWu^tihn^J^ D 

^Kt}^^<D^y U-Afc>P}-r.-rSB#^iJ ( t m 1 . t m 

2. ) StCtf ^ C<hJCj;«:,t:. m \ iiiCTTcrmx 

ti. B^glJt ni3tC*5l>rj[:^Jg "Silence" iPhif, 
^ tcaSU. H#Mt m5iC*5l^rt^l^ "n 

t^hi-m "n2" tCilj^U. ^Mtm9tCtet,:»ri^^ 

2'' t^hitm 3 " icS^U. B^M t m 1 1 ^C:fe 
X^^Xifm "ri3" 7&^^tfc« "a 1" CC^^^b/cJ: ^^C^ 

[0047] [2-2-3-2. v W-A#iacD>^j;t:]± 



msoy u-M.^^^MT^c^t}^x^^J:'>iiCrj:^xi> 

A:fy^^<DMft^T^^m^moy\y-^m±^. ^y\y 
30 [ 0 0 4 8 ] a 1 1 ^cm-tmx\'x. ^^a^^oH^M 1 1 

1 --^ t t 3 (Dy " S i 1 e n c e " iC*tl£ 

B#Mt 1 1 0-'<7>:7U-A}05W* "a" icS^jSLr 

i^'So --:^r. i^^';^tfd?-fc:T;i/=fyXA^cj:orA;^ 

*^co«M^#;0^?*^?ti. A*«^^DB#^J 1 ml - t 
m2CD>' "Silence" ^cJ^iSL. H# 
Ml m3--l m I 0CD:7 U-A:^^^* ^n" ^c>f^J[£L. 
B#Mtml 1 --(D>^ "a" 0C>(tlSOri^ 

■So ^Ur. "Silence" fC>f*lST-S:? 

40 Acbur. A:f?^j^coB#giI t TTi 1 (D:7 u-A<hS+a^^ 
(Dfff^fJ t t 1 1^5 -it. A:^?#^<Dff#^J t 

m2c7:)-7 lx-Ai>^a#^OB#SiJt 1 2cD:7l/-A^— 
S^it-So A;tlW^COB$SlJ t m 3 iC:foC^rt^:M " S i 1 
en c e" iPh^ "n 1" CcjS^Lri^acD-C. 
"n" Cc^lS1-S"7 U^-Aibr^J. A:^#^cDBf^Jt 
m3(D:7V-A;0^|7JCr)>' U-A^C^cao 

MMidM^cXfiW^Mt t 4CD::7 \y-2^tf>hx^^<0 
50 ^^Jt t Atrj:^ ms : Xt^-:. :7'S 5 — S 7#M) o 



CIO) 

17 

AB#raKPS^2s6t: (MS : :^f^yy'Sb^S 1 1# 

[0049] id^tr. 01 1 ec^-rt^Jrci. A*r^?^ 

OB#^Jt m3^ tm 1 0 *"CCD 8 :7 A^/^?^^^ 10 
"n" ^c*fjSL.rc^>6:7 u-A(^^Mt 14-119^ 

[0 0 5 0 ] [2-2-3-3. mf-^tcommitc 

L.r:feO . :^7'7:*-yi^a&3:M I D I f'-^tcS':5l=»r^ 

mf-^^ci^^tifmmmm (0mt. m i d i 
m-^i.^x. mm(Dmnmm^^fS.Lxr'y^^>hmi 

tf^i^miisHfcmimmt^-^yvyu-Amnmm 

M^cmm^tixi>^^mmyt^nt^m^Lx. mm 40 

[0 0 5 1 ] ^/c. mi 2i^cmr j^'yt^m^mm^f 
(I t„ - 1. 1 ) ^m\^>x. mm^mmi^xmmm^m 
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[0 0 5 2 ] [3, mm']-:^mmt. iMi^tcmmm 

[0 0 5 3 ] [3 - 1 . '^m(omz^w^\^']±.Mwmmx 
1 ^• X e ^ - T 3 y X A ^ffi I ^x^mm^'^ 
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